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INTELLIGENT REMOTE MULTIMODE SENSE AND DISPLAY 
SYSTEM UTILIZING HAPTIC INFORMATION COMPRESSION 

S Background of the Invention 

The present invention relates generally to a system and method of 
providing haptic (position, orientation, force, & tactile) feedback to the 
human operator or a remote or virtual environment. 

Devices and systems are well known for measuring or monitoring one 
10 or more characteristics of an anatomical part (see for example U.S. Patent 
Nos. 4,986,280, 3,258,007 and 3,364,929). A human controlled position 
sensing device for use in the field of robotics is also well known (see for 
example, U.S. Patent Nos. 4,328,621, 4,534,694, 4,608,525, and 
4,674,048). 

15 

Force Feedback Systems 

Systems are also well known for providing force feedback to human 
operators. Ftfrce-reflective devices exist in a variety of forms, from 
joysticks to arm and hand controllers. These devices serve as input devices 

20 and haptic interfaces for virtual environment manipulation, or teleoperation. 
A significant amount of research has been conducted on robotic hands and 
force-reflecting controllers for robotic arm manipulation. While a few 
controllers were anthropomorphic, most were typically tailored to imitate the 
slave robot's kinematics. The advantage of anthropomorphic controllers is 

25 that they imitate the kinematics of the human hand and arm, thus possessing 
the largest range of human operator (user) motion. Yet these past 
anthropomorphic designs were unable to achieve high quality force feedback 
while retaining minimal actuator size, a requirement for system portability. 
So while many forms of force-feedback devices have been developed, few 

30 completely anthropomorphic systems exist. The systems tended to he 

massive and bulky , and required large centralized computational resources 

I 
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tightly coupled to the sensory system, further preventing an -efficient 
integration of the system to the human operator. Finally, the systems were 
non-modular, requiring extensive work to add additional haptic sense and 
display devices. 

5 

Tactile Systems 

Tactile displays have been the subject of much research activity in the 
past thirty years. While neuroscientists studied the physiological roots of 
human tactile perception, engineers were interested in tactile display devices 
10 that substituted or augmented other senses such as sight and hearing. Tactile 
displays also demonstrated their usefulness in virtual reality simulations and 
telerobotic operations. 

A system utilizing tactile display devices provides the teleoperator 
with additional information, such as contact locations, that could not be 
15 adequately displayed by force alone. Tactile displays also simplify the 

requirements placed upon force reflection. A combination of tactile display 
and force reflection is haptically more effective, technically less -challenging 
and economically more feasible than force reflection alone. Therefore, a 
* high performance force reflective exoskeleton master should incorporate 
20 tactile displays. 

Tactile display technologies have slowly evolved in the past thirty 
years. Many different tactile display technologies are available today. *For 
instance, air or water jet displays are used to trace outlines on the skin and 
are attractive because they eliminate the need for moving mechanical parts 
25 and because they exhibit a consistent frequency response. Air or water 

bladder displays are used to apply a continuous pressure on an area of skin 
but suffer from extremely low bandwidth. 

Mechanical tactor displays include most of the vibro-tactik devices 
described in the literature in addition to a few displays operated at a very 
30 low frequency that would not normally be considered vibrotactile. A few 
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single-element display devices exist but the vast majority of mechanical 
tactile displays involve tactor arrays. The number of elements in these 
arrays run from two to 400. Almost all of the devices in this group involve 
indenting or impacting the skin perpendicular to the skin surface. Two 

5 methods of actuation dominate these displays; solenoid activated pins and 
cantilevered piezoelectric bimorph reeds with a pin mounted to the tip. 
Another method of actuation that has great potential is the use of shape 
memory alloys. As a current is passed through the alloy, it changes shape 
and deflects a spring that holds a tactor pin in place. Voice -coil displays 

10 generate vibratory sensations by mounting a voice coil against the skin. 

Electrotactile displays elicit touch sensations by indiscriminately 
stimulating all the touch and pain sensors in the skin with a current passing 
through the skin. These devices usually involve a remote stimulator (which 
supplies the power and generates the stimulation waveform) end a number of 

15 electrodes mounted on the skin using a sticky dielectric. -Each electrode pair 
constitutes a stimulation site. While some systems involve single or a small 
number of isolated electrode pairs, arrays of -electrodes were also popular, 
with number of elements running up to 1024 in one instance. 

In all of these devices, the sensing or feedback element contains no 

20 intelligence or ability to interact with human stimuli itself. They are hooked 
to a computation engine which contains a world model which interprets the 
raw sensory information and then sends low level commands for behaviors 
to be achieved. There are many limitations in this approach which have 
been apparent and much discussed in the research literature (papers on force 

25 feedback in the presence of time delays by Sheridan and others) and 
elaborate methods of modeling system performance to overcome these 
problems. The more complex the feedback, the more extreme the problem 
becomes. In fact this has led the virtual reality research community in the 
opposite direction from this invention, to use larger and more powerful 

30 computational engines and very simple feedback modes with limited 
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performance success. Most of the systems used are on non real-time 
computation engines which makes the problem of providing stable, realistic 
and predictable feedback even more difficult. " 

The current invention relates to a method and system for overcoming 

5 these limitations through development of a architecture which provides local 
intelligence and computation coupled with a method of compressing haptic 
information through various protocols which reduce the amount of 
information that is shared between the world modeler and the haptic 
system. System speed is improved and performance is enhanced making it 

10 feasible to employ multiple modes of haptic feedback simultaneously without 
overloading the communications link or basic computation engines. Since 
multiple modes of sensory feedback are utilized to reflect the real 
interactions between a human body part and the outside world, a realistic 
user interaction with a.virtual or remote environment is achieved. 

15 Therefore, an object of the invention is to provide local processing to 

each sense and display device so as to provide real-time feedback to the 
human operator. 

Another object of the invention is to provide a modular framework in 
which to extend the sensory feedback capabilities as required by the human 
20 operator. 

Another object of the invention is to minimize the communications 
bandwidth required between the haptic system and the world modeler . 

Another object of the invention is to provide an anthropomorphic 
system while retaining portability. 
25 Another object of the invention is to provide a system which can 

sense and/or display force, slip, vibration, shape, stiffness, friction, texture, 
pressure, mass, temperature, sound and electromagnetic radiation to a 
human operator. 
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Additional objects and features of the invention will appear from the 
following description in which the preferred embodiments is set forth in 
detail in conjunction with the accompanying drawings, " 

5 Summary of the Invention 

A system for sensing and displaying sensory information to a human 
operator comprising a world modeler adapted to receive sensory input from 
the operator over a communications medium. The world modeler modifies 
either a virtual world which is a mathematical model of an environment, or 
10 modifies a real world which is a remote real environment. The world 
* modeler then sends sensory information from the real or viraial 
environments over the communication medium to the haptic system. The 
haptic system displays this sensory information to the human operator. The 
result is that the operator is able to manipulate a real or virtual environment 
15 remotely through the information sensed and displayed by the haptic system. 
The haptic system is composed of sensory modules. The sensory 
modules are further composed of a controller (i.e. CPU, DSP or 
microcontroller unit), communications unit, and sensory and/or display 
devices. 

20 The display devices can include heating elements to convey 

temperature, voice coils to convey vibration, goggles to display visible, 
motors to display force, brakes to display friction, speakers to -convey sound 
and tactile sensors to convey various senses of touch. 

The sense devices can include encoders to convey position, strain 

25 gages for force, display device feedback to measure torque, thermocouples 
for temperature and cameras for vision. 

Sensory information is compressed through various methods. One 
method is to transmit Only those sensory cues required to convey the 
intended sense. Another method is to only select the sensory information 
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required for a task. Further, the data selected may be compressed further 
through conventional data compression techniques (i.e. JPEG for video). 

The complete system may be scaled as needed. Minimal systems may 
only convey tactile information through a haptic hand . Larger systems may 
5 utilize exoskeletons to convey joint information and force to the system. 

Brief Description of the Drawings 
The invention is described and explained in more detail below using 
the embodiments shown in the drawings. The described and drawn features, 
10 in other embodiments of the invention, can be used individually or in 
preferred combinations. The foregoing and other objects, features and 
advantages of the invention will be apparent from the following more 
particular description of a preferred embodiment of the invention, as 
illustrated in the accompanying drawings. . 
15 FIGURE 1 is a block diagram of the present invention, 

FIGURE 2 is a block diagram of the world modeler of the invention. 
FIGURE 3 is a block diagram of the haptic system of die invention. 
FIGURE 4 is a block diagram of variation of sensory modules of the 
invention. 

20 FIGURE 5 is a block diagram of the invention as implementing 

teleoperation. 

FIGURE 6 is a front view of the hand embodiment of the invention. 
FIGURE 7 is a side view of an example tactile sensory used to 
convey slip, 

25 FIGURE 8 is a block diagram of the haptic system for the hand 

embodiment of the invention. 

FIGURE 9 is a flow chart of the control software for the hand 
embodiment of the invention, 

FIGURE 10 is a front view of the hand, arm and vision embodiment 
30 of the invention. 
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Description of the Preferred Embodiment 
The current invention relates to a method of overcoming past design 
limitations through development of an architecture which provides local 
intelligence and computation in conjunction with compression of die haptic 

5 information through the use of protocols which reduce the amount of 
information needed to be shared between computation engines at a given 
instant. By utilizing local processing within the sensory units the system 
speed and performance is enhanced making it feasible to employ multiple 
modes of haptic feedback simultaneously without overloading the 

10 communications channel or basic computation engines. Multiple modes of 
feedback are necessary in order to have a realistic interaction between a 
human body part and a virtual or remote environment. 

The present invention generally relates to a system architecture for 
providing sensory feedback in general to the human operator. In particular, 

15 touch feedback is necessary for interactions with the human operator in 

controlling either a computer generated, virtual environment or a real remote 
robotic environment. 

Sensory Display and Sense 

20 The sense of touch, or haptic sense, is complex. Unlike vision, touch 

involves a direct physical interaction, so that human actions cause and 
change what is perceived. It is the fundamental role of touch to sense the 
results of these contact interactions in order to guide manipulation and body 
motion. The current art only contains devices for presenting a crude tactile 

25 image to a location on the body. This type of information represents only a 
fraction of the total information sensed by the human sense of touch. Many 
different types of basic information about the contacts are extracted by 
people through a combination of four different tactile sensory systems, 
measurement of joint motion, and measurement of joint torques . This basic 

30 information can be organized broadly into passively acquired information 
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and actively acquired information. Passively acquired information can be 
. gathered without relative motion between the body and an object and without 
varying the forces between the body and the object. Actively acquired 
information requires either relative motion or varying contact forces. A 
5 summary of some of the basic types of cues or sensed information is given 
below. 

Passively Acquired Identification Cues 

Passively acquired identification cues can be obtained by the act of 
10 holding the object in the hand. Three major cues have been identified in this 
category: 

1 . Determining the local contact normal and curvature of the 
surface at each contact. 

Determining the local contact curvature and labeling it as a 

15 point, edge, or planar contact is important for grasp acquisition and 

object identification. Different contact types have different grasp 
properties which affect stability; Salisbury, J.K. and Mason, M.TV, 
Robot Hands and the Mechanics of Manipulation, MIT Press, 
Cambridge, MA (1985). in addition, the local contact normal and 

20 curvature provide a strong pruning heuristic rule for identifying 

objects and object pose; Grmson, W.E.L. , Object Recognition by 
Computer: the Role cf Geometric Constraint, MIT Press, 
Cambridge, MA {1990). Local object curvature can be measured 
passively be examining the normal contact force distribution, 

25 2. Determining the surface texture at each contact. "Surface 

texture affects grasp stability. Rough textures are -generally easier to 
grasp than smooth textures. In addition, surface texture can also be 
used as a pruning heuristic rule in identifying objects and object pose. 
Texture cues are produced both from the spatial distribution of the 
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contact force and the dynamic vibration -effects produced at the skin 
during motion. 



5 3. Gross object shape. 

By using the fingeir geometry and the location of the contact 
points in the hand, the shape of the object can be estimated. For 
example, an initial grasp might be a power grasp in order to get a lot 
of contact information about an object. X3nce the object and its shape 

10 is identified, a dexterous grasp might be used for manipulation. 

Actively Acquired Identification Cues 

With some local active exploration, properties relating the reaction 
forces to the applied force can also be determined: * 
15 l. Determining the local stiffness at each contact. 

The local surface stiffness can be estimated by applying 
varying normal forces and measuring the change in contact area. 
Softer surfaces will produce a larger contact area for a given applied 
force . This is not a significant cue for manipulation of -control 
20 surfaces and tools since almost all of the objects involved are rigid* 

2. Determining local fractional properties. 

The local friction along with the local contact type control the 
local grasp stability. Friction can be measured by applying varying 
tangential forces anil then detecting the onset of slip {slip detection is 
25 a separate basic manipulation cue; see next Section). This task also 

requires the ability to measure tangential forces. 

3. Overall object stiffness. 

By integrating the contact force over an area with the joint 
torque's, and correlating this with joint deflection, the overall stiffness 
30 can be computed. 
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4. Detecting slip. 

Detecting the onset of slip between an object and the hand is 
essential for grasp maintenance. Slip detection is used to determine 
the necessary grasp forces at each -contact during all stales of 
5 manipulation. 

5. Determining object mass (weight), center of mass, and 
moments of inertia. 

By manipulating a grasped object and measuring the resulting 
joint torque's and net contact forces at the hands for different 

10 configurations, the weight, center of mass and moments of inertia of 

the object can be computed. TMs information can be used for object 
identification and pose determination, Siegal, P.M., Pose 
Determination of a Grasped Object Using Limited Sensing, Ph.D. 
Thesis, Department of Electrical Engineering, MIT, Cambridge, MA 

15 (1991), as well as for computing the necessary torque's for throwing 

or manipulating the object. 

6. Estimating directions of contact constraint. 
Assembly is the process or bringing a moving part into a 

constrained relationship with a fixed part. In order to control the 
20 assembly process, the directions in which movement is constrained 

need to be estimated. Contract constraints are estimated using 
measurements of the reaction forces as a function of position and by 
measuring the direction of impact forces. 

7. Detecting changes in contact constraints . 

25 This is one of the most common tasks during manipulation. 

The detents in switches, the termination in screws, the impacts from 
mating two parts are all examples. The onset of the change can be 
detected by looking for unexplained impact forces and the direction of 
the impact force. 

30 
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Current tactile feedback devices present a fiat contact image using 
either arrays of tactors or raster scan techniques. Appendix A includes a list 
of prior art devices whose teachings are herein incorporated by reference. 
This type of display can only be used for very local shape display or 

5 textures, and is limited to a flat type of display. A need exists for devices 
for displaying a variety of the remaining cues, such as those described 
above, (slip, contact curvature, gross texture) and those cues such as 
vibration, fpr which devices already exist in a realistic and natural fashion. 
Thus the object of the current invention is to provide a modular structure 

10 under which individual haptic cues or multiple cues can be effectively and 
efficiently presented to the human operator. 

ffaptic Compression 

The bandwidth required to completely characterize and display haptic 

15 feedback of all senses would normally be prohibitive. Various forms of 
haptic compression are used to limit this bandwidth. 

Haptic compression is based upon human perception. The 
compression algorithms ask the question; what information has to be 
presented to perform this manipulation or recognition task. Haptic 

20 compression is based in sending a set of physical parameters and a number 
indicating the model type to the haptic display computer. Locally, the 
model number is used to pull up a stored model of a component of the task 
physics from the memory of the local haptic processor . The parameters then 
control the specific instantation of that model. 

25 For example, perceiving a fine surface texture is done primarily 

through perception of spatial frequencies through vibration. By measuring 
relative velocity and having the spatial frequency stored as a two directions 
fourier spectrum the texture display would be able to simulate a complex 
texture. Compression occurs because (1) spatial representation does not 

30 resolve all surface features; (2) vibration is substituted for an array display. 
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Curvature and surface normal is based on aft analysis of the task 
which shows that only this information is necessary for performing the 
required manipulation task. The computer sends 2D curvature properties 
and 1 contact point to represent the surface to the haptic display computer, 
5 The display then takes care of bending up the surface. This is a first order 
representation of the more complex shape. Since not all the information is 
shown, there is information compression. 

For local stiffness, the haptic processor gets a number K which 
represents< the contact stiffness. Based on this number it simulates a surface 
10 of stiffness K by applying the correct forces given a displacement relative to 
the display. 

Like local stiffness, a slip haptic processor only needs the coefficient 
of friction mu. The slip processor can then measure the normal force and 
the tangential force to determine the current state of the slip feedback. 

15 In order to implement general constant constraints a set of constraint 

equations C_i(g,x) are stored on the local processor. The constraint 
parameters g and the specific model type i are sent to the local processor in 
order to instantiate a specific constraint model. The local process applies 
corrective forces to try and force the coordinates of the display, x, to obey 

20 this equation in the display . 

Information may also be compressed through the removal of 
extraneous information. One method is to include only those senses 
appropriate to the task. A second method is to use psychophysical 
compression, i.e. only including the sense cues necessary to give the 

25 operator the ability to "feel" the characteristic being displayed. A third 
method is to use data compression prior to transmission across the 
communication channel. A fourth method is data thinning, ^e.g. 
communicating only the changes since the last transmission or transmitting 
only when changes are large enough to warrant. A fifth method is to 
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partition the system such that the computation and display subsystems share 
a lower bandwidth communication channel. 

IK 

Sense Selection 

5 There may be no need to include temperature display , if the task is 

one of simply grasping and moving an object of known temperature. There 
may also be no need to transmit with absolute fidelity all hand joint torques 
if the application requires only the input of the thumb and forefinger for 
acceptable results. Selection of the minimal number of senses required for 

10 convincing haptic feedback is the primary method of reducing 
communications traffic. 

Psychophysical Considerations 

Studies have shown that the human may only need a subset of sense 
15 cues to synthesize the sensory feedback. Sense-mapping, or replacing one 
sense for another, can be used for feedback of senses or other data sources 
otherwise difficult to display. An example is using a vibrotactile display to 
display surface temperature through variations in vibration amplitude. 

20 Data Compression 

Various methods exist for brute-force data compression. The least 
efficient algorithms, on a percentage compression basis, offer the ability to 
completely and accurately reproduce the input data after decompression. 
More efficient methods, such as fractal compression, reproduce the data 

25 with reasonable fidelity and require relatively small transmission bandwidth 
but require some knowledge of the nature of the data being compressed. 
The best protocol will depend on the psychophysics of the senses being 
displayed, 

30 Data Thinninff 

. : * 13 
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Sending a continuous stream of absolute sensory data across the 
conununicatipns channel may be inefficient. Once the minimum bandwidth 
of the sense displayed is established, only state changes may need to be 
communicated. The channel could become event driven by transmitting only 



System Configuration 

The system may be partitioned so that there is a reasonable amount of 
computation performed by the display driving electronics. Each sense can 

10 have it's own processing, sharing and communicating a common knowledge 
of the "world" but reacting in ways unique to the^ense being displayed. 
This approach is readily scalable due to its modularity. For example, the 
visual display hardware may be suited to fast graphics rendering but does 
not need the ability to compute other sense data. The only need is to be 

15 able to modify and display it$ world view given input data from other 
transducers and senses. 

The prototypical haptic module would consist of a processor, 
memory, power and drive electronics, transducer interfaces, and a 
communications interface and controller. The software on board the haptic 

20 processor would include the kinematics mapping the sensor data to the 
display. If, for example, finger force feedback was desired, the module 
would query the finger's position sensors, map this data to joint angles and 
finger geometry in it's world view, sense collisions with objects in it's world 
view, and calculate resultant forces. It would communicate the updated 

25 world view to the other modules and apply the appropriate torques to the 

finger, knowing the configuration of the force feedback device. The system 
is tolerant of system communication latencies as the feedback loop is local to 
the haptic processor. 
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With the sensors and transducers available, the option of mapping one 
sense to another becomes feasible and perhaps desirable in some cases. Hie 
ability to map physical parameters not normally sensed <for "example 
electromagnetic field strengths) to human senses is compelling and may even 

5 lead to more intuitive understanding of complicated phenomenon. Sense 
data may be freely scaled, such that a remote robot manipulator could 
experience forces on the order of tons that could scale to ounces on the 
haptic display. At the other end of the spectrum, mkroforces can be scaled 
to pounds at the haptic display, allowing micromanipulation with extremely 

10 fine resolution of control. 

Generel System ArcfritCTre 

Referring now to the drawings, wherein like numerals are used 
throughout the several views to identify like elements of the invention, there 

15 is disclosed a system for remote multiinode sense and display as shown in 
Fig.l. The haptic system 14 interacts with a user 16* Sensory input 19 
from the user 16 is collected by the haptic system 14. This information is 
compressed by the haptic system 14 and transmitted onto the 
communications link 12 to the world modeler 10. The world modeler 10 

20 uses the information to manipulate either a real or virtual world. The world 
modeler monitors and measures the state of the world, transmitting display 
information onto communications link 12 to haptic system 14, where the 
display information is converted to sensory output 18 for the user 16. 

With reference to Fig. 2, the world modeler 10 of Fig. 1 may take 

25 two forms 10a or 10b. The first form 10a utilizes a computer system 20 to 
create a mathematical model 22 of a virtual world. The second form of 
world modeler 10b utilizes a computer system 24 to sense information 25 
and send control information 26 to remote input-output system 28. This 
remote input-output system 28 exists in a real environment. 
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The haptic system 14 of Fig. 1 is further Composed of at least one 
sensory module 30 as shown in Fig. 3. Each sensory module 30 may take a 
variety of formats. Fig. 4 shows examples of sensory modules 30a, 30b and 
30c. A sensory module is composed at a controller 40a communications 
5 unit 44 and at least one sense unit 42 or display unit 43. Sensory module 
30a contains only one sense unit 42, sensory module 30b contains only one 
display unit 43, while sensory module 30c contains two sense modules 42 
and one display unit 43. 

Sense unit 42 contains sensors to monitor sensory input from the user. 

10 For example, this sensory input may be force, position, pressure, 

temperature information as experienced by the user. Display unit 43 
contains displays to convey information, such as force, temperature, 
vibration, texture, friction, shape and stiffness, to the user. The result is the 
user interacts with either a remote or virtual world through sensory input 

15 controlled by the haptic system. 

Fig. 5 shows the system implementing teleoperation. A user 56 
interacts with the remote world 50. The haptic system 14 -gathers sensory 
input 52, This information is compressed by the haptic system 14 and 
transmitted on communications unit 12 to the world modeler 10. The world 

20 modeler generates control information 26 from the compressed sensory input 
received from the communications link 12. The remote input-output unit 28, 
which may be a robot or mechanical arm, responds to the control 
information 26, thereby acting in the remote environment 50. The world 
modeler also takes sensory input 25 sensed by the remote input-output unit 

25 28 from the remote environment 50. This sensory input 25 is compressed 
by the world modeler 10 and transmitted onto communications link 12 to the 
haptie system 14. There the compressed sensory information is displayed 54 
to the user 56. The result is that user 56 controls a remote input-output 
device 28 in a remote world '50. 
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Wand Haptic System 

The hand haptic system is comprised of touch sensory modules which 
are composed of a local processing element connected to display and sense 
hardware via wire or fiber optic connection. The sensory module |is in turn 
connected via a wire or fiber optic link or a communications link such as an 
ethernet port, serial port, bus connector such as a Bit 3, parallel port, game 
port, a local area network to the world model transmitting either direct 
information for simple or low speed applications or information which has 
undergone haptic information compression. The human operator controls 
the world model through interactions with the visual model information 
(displayed on a computer monitor) and the touch display hardware. 

This results in the touch display hardware reacting and displaying 
touch information to the operator. The touch sensory module, because it has 
a local processor system, reacts directly to the human stimulus and thus can 
interact at the high speeds required for the feedback to feel realistic even in 
the face of rapidly changing stimuli from the human operator. This reaction 
can be completely independent from the world model and visual display and 
thus not be slowed by a slow communications link and can therefore run at 
speeds upwards of IKHz if necessary. This is made possible because of the 
architecture of the local processor unit. This processor can be a single -chip 
such as a DSP or Microcontroller, a board with multiple chips or a 
computer, depending upon the number of sensor and display elements 
contained in the touch display hardware and the complexity of the model. 

The haptic compression is achieved by utilizing the basic research into 
human perception of touch described above to determine which touch-cues 
are being utilized for a given type of manipulation. For example, when an 
object is grasped the joint torque cues as well as slip information are used to 
ensure that the object is stably grasped. The algorithms in the local 
processor determine the type of cues required and their magnitude thus 
reducing the amount of information which needs to be transmitted to the 
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other system components. The world model only needs to know the 
resulting object status (i.e. position and orientation, or forces acting on the 
object, depending upon how the world model is structured)' thus the 
information content of the transmitted signals is greatly reduced thus 
5 improving the overall performance of the system. 

In the embodiment shown in Fig. 6, three sense and display elements 
were used; slip on the finger tips, force feedback to the index finger and 
thumb, and vertical resistance. In addition the device provides position and 
orientation of each of the system elements, 

10 In this embodiment a simple slip display, shown in? ig. 7, was 

developed as disclosed in U.S.S.N. 08/187,646, entitled "Multimode 
Feedback Display Technology " filed January 27, 1994 (herein incorporated 
by reference). The slip display consists of a delrin cylinder 71 with a 
textured surface, directly mounted on the shaft of a miniature gear motor 72. 

15 The cylinder is supported on the other side with a miniature bearing 73. 
The motor is held by a set screw in the mounting bracket 70. 

In Fig. 6, the slip display 61 was attached to the tip of each of the 
index finger force feedback apparatus 62 and the thumb force feedback 
apparatus 63. Force reflection to the index finger is provided by a modified 

20 two-degree-of-freedom SAFIRE (Sensing and Force reflecting Exoskeleton, 
Patent application serial #07/961,259 submitted 10/ 15/92) prototype, which 
consists of two cable-driven linkages 65, with the two motors $6 mounted on 
the forearm. The axes of rotation of the linkages are -co-located with the 
PIP and MCP joints of the' index finger. Force reflection to the thumb is 

25 provided by a one-degree-of-freedom device 67, also driven by rabies 69 
from a remote motor mounted on the forearm 68. 

The vertical resistance and position and orientation in space is 
provided by a boom 64. The vertical force reflection axis provided up to 2 
lbs of virtual weight at the hand. It also supported the weight of the 

30 SAFIRE through counterbalancing. The stiffness of the vertical boom up to 
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the SAFIRE attachment bracket exceeded 140 lb/in, the minimum stiffness 
required for a cantilevered beam to feel rigid to a human operator. The 
resolution of the vertical and horizontal position sensors for the hand and 
index finger was 0.005 inch. Two linear ways were mounted on opposite 
sides of the vertical support structure, which was rigidly attached to a heavy 
base. 

, The drive mechanism 201 for the vertical degree-of-freedom was 
located on the back of the support structure 200. The motor providing 
vertical force reflection was horizontally mounted on the slider riding in the 
rear linear way. A threaded capstan was directly mounted on the shaft of 
the motor. A 7x7, 0.032 inch diameter steel cable was wound 3-4 turns 
around the capstan. One end of the cable was terminated on a cable 
tensioner at the top of the structure. As the motor rotated, the cable hoisted 
the slider up and down the rear linear way. s 

The rear slider was connected through a second cable and a set of 
pulleys to a plate mounted on two sliders riding in a second linear way, 
mounted on the front side of the vertical support structure. Using two 
sliders instead of one greatly improved the twisting and bending load 
capacity of the front linear way . The two sliders were mounted on the plate 
through two pivots to minimize bending moments applied to the front linear 
way. 

Vertical forces were transmitted to the SAFIRE by a horizontally 
mounted linkage. The linkage consisted of a distal block attached to the 
front slider plate, proximalblock, and a 12 inch long rod. The distal block 
housed the bearings and the encoder for the first passive pivot, also called 
the distal pivot. One end of the rod was pivoted about this block so that the 
linkage rotated freely in the horizontal plane. The other end of the rod was 
pivoted about a second bearing/encoder housing, also called the proximal 
pivot. The proximal mounting block was attached to the SAFIRE base with 
a rigid bracket. The length of this bracket was designed to position the hand 
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so that the proximal pivot was between the tip of the index finger and the 
thumb. 

The motor for vertical force reflection was selected Based on the 
power requirements. Ideally the motor should produce a force of 2 lbs at 
5 the hand. It is estimated the hand could move at a maximum vertical speed 
of 8 in per sec. We selected a 90W motor without a -gearhead to provide a 
smooth operation with minimal friction. This is designed to produce a force 
that exceeds the corresponding frictional force, if the frictional -coefficient 
between the finger and the display is estimated at 0.5, the tactile display 

10 must be able to produce 1 lb. of force. If the display moved past the finger 
at 1 in/s, the required power would be around 0,1 12W. A motor producing 
0.25-0.5W would therefore be a suitable actuator for this display. The 
MicroMo Series 1016 motor fits these requirements; it is 0,384 inch in 
diameter, 0.63 inch long and weighs 0.23 oz. This is mounted in die space 

15 beyond the fingertip. 

The hand haptic system is interfaced to the world modeler, in, this 
case implemented on a graphics rendering PC, through a local processor 80, 
in this case a VME card cage which houses a TI C30 DSP board 81, I/O 
boards 82, 83, and 84, and a BiT3 VME-ISA bus adapter 85. 

20 The physical simulation and device drivers reside on a TI<30 

floating point DSP board 81. The DSP board is installed in a multislot 
VME cage which also houses the I/O electronics. The simulation is 
interrupt driven with a time step of 300 Hz, which is the minimum 
performance level necessary for a stable and realistic simulation. jligher 

25 performance could be gained by parsing out the simulation tasks among 
several DSP processors and executing them in parallel. Vot example, 
communication tasks with the graphics PC and boom position sensing could 
be handled by a different processor than the main simulation loop. 

The DSP simulation, as shown in Fig. 9, communicates with the 

30 graphics PC via a BiT3 VME-ISA bus adapter. This high speed 
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communications channel handles graphics information generated by the 
simulation and simulation parameters generated by the user interface. The 
user interface starts the standard simulation demonstration and provides 
program status information. Since the user interface and graphics rendering 
5 can be somewhat asynchronous to the simulation, graphics and PC system 
latencies will not disturb the simulation which depends on a uniform time 
step. Also the bandwidth required for this information is low enough to not 
require a dedicated processor. This software approach does not require the 
simulation and graphics to communicate a significant amount of objects 
10 characteristics upon startup of the simulation so that a minimum of data is 
exchanged during run time. Data from the simulation need not be 
transmitted to the graphics PC any faster than the graphics refresh rate. For 
example, at a 60 Hz refresh rate, new data would be needed at a m a xim u m 
of every third integration time step. • 
15 Sensing the joint .positions and orientations and applying appropriate 

forces and slip requires additional I/O electronics. The joint positions are 
sensed with digital position encoders . The joint torque's and slip applied to 
the touch display are voltage controlled by the simulation via 6 12-bit digital 
to analog converters and external current mode motor amplifiers. The 
20 encoder counters and digital to analog converters reside on the high-speed 
VME bus with the DSP board and are directly accessed by the DSP. The 
generality of the control electronics and overall architecture mates the 
system adaptable to other force reflecting masters. The minimum 
configuration can handle six output and -eight input degrees of freedom, and 
25 can be easily expanded to any reasonable number by adding I/O modules 
and, if necessary, DSP processors. 

Almost all of the application graphics and DSP code is written in C. 
The simulation can be developed on the graphics PC and debugged with 
tools available for the PC/B1T3 development environment. Porting the 
30 simulation to the DSP is then relatively straightforward. In fact, 4he existing 

21 



WO 95/20788 PCT/US95/01227 

simulation can be debugged on the PC and then recompiled for the DSP with 
only minor changes. Hardware based debugging tools are available for the 
DSP v although the DSP/PC interface provides a relatively powerful 
stethoscope for all but the most low level debugging or obscure 
5 compiler/hardware bugs. The DSP executable and source code resides on 
the PC and is downloaded to the DSP at run time. 

This system and method can be applied to a wide variety of physical 
embodiments including simpler systems and more complex and capable 
systems with more feedback modes utilized. 

10 

Hand/Arm/Vision Haptic System 

An alternate embodiment utilizes the hand haptic system in 
conjunction with an exoskeletal arm mechanism combined with a vision 
display. The exoskeleton arm consists of two interconnected. mechanisms, 
IS one for the shoulder and the other for the elbow and forearm as shown in 
Figure 10. 

The shoulder mechanism 90 consists of a three DOF motor driven 
gimbal, a passive prismatic joint (spline), a 2 DOF passive gimbal and a 
upper arm cuff. The elbow mechanism 92 is connected to the shoulder 

20 mechanism 90 at the distal end of the spline 91. This allows the system to 
accommodate changes in die distance between the shoulder 90 and elbow 
mechanisms 92. The spline also eliminates the need for adjustment for 
different upper arm lengths. 

The center of the distal gimbal 93 is located on the axis of the upper 

25 arm E/I rotation. The elbow F/E motor 94 is pivotally jointed to both the 
upper 95 and lower 96 cuffs. The final step was to use a brake 97 at the 
distal gimbal of the shoulder mechanism to ground the elbow/forearm 
mechanism. 
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The elbow mechanism has two DOF. The first DOF is aligned with 
the axis of elbow F/E. The second DOF is placed along the forearm with a 
remote center drive mechanism. 

The three motors 100, 101, and 102, at the shoulder gimbal can he 

5 coordinated to apply a force at the upper arm which is approximately 

perpendicular to and intersects with the longitudinal axis of the upper arm. 
This force creates a torque about axes of the shoulder F/E and Ab/Ad. The 
motor at the elbow controls the Flexion/Extension torque about the elbow. 
The rotational torque about the longitudinal axes of the upper and lower 

10 arms are controlled in two different ways depending on the position of the 
elbow joint. When the elbow is extended the motor at the wrist applies 
rotational torque to the longitudinal aces of both the upper and lower arms, 
since their motions coincide. When the elbow is flexed, the motors at the 
shoulder can be coordinated to apply a force at the forearm cuff to produce 

15 a torque about the upper arm E/I, and the RCD applies a torque to the 
forearm. 

The biggest advantage of this design was that it conformed to the 
anatomic motions of the entire arm. It imposed no kinematic constraints on 
the arm because the shoulder mechanism had full 6 DOF and the elbow fVE 

20 mechanism formed a 4-bar linkage with the elbow F/E joint. Therefore, it 
was very comfortable to wear. It also offered large Range Of Motions 
(ROM) . It not only applies forces to the entire arm grounded to the back* 
but also torque to individual joints. The former allowed us to simulate 
picking up an object by bracing between the forearm and the upper arm or 

25 between the entire arm and the chest. The latter could simulate contacts of 
objects in a virtual environment by any portion of the arm such as hand, 
forearm and upper arm. In summary, this mechanism offered all the force 
display requirements for a force reflective arm master and conformed to the 
anatomic motion of the arm. 
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A higher performance version can be achieved by using magnetic 
brakes in conjunction to tow miniature motors or actuators. The two 
magnetic particle brakes may be used to replace the first two motors at the 
shoulder gimbal. In order to use brakes, the mechanism must be fully 

5 balanced about these two axes because brakes are passive and cannot lift. In 
addition, the miniature actuators or motors must be put in series with, and 
parallel to, the two brakes for active force control. One way to achieve this 
is to activate, the 2 DOF passive gimbal at the upper arm cuff with such 
miniature motors. With the brakes locked, the two miniature actuators or 

10 motors can display small forces to the operator at very fine resolutions. To 
simulate a hard contact, simply lock the brakes and drive the two miniature 
motors to their respective hard stops so that the braking forces can be felt by 
the upper arm. For a force in between these two extremes, the -combination 
of Pulse Width Modulations (PWM) on the brakes and force controls oh the 

15 miniature motors can be used simultaneously. 

The video sense is accommodated via a head mounted display or 
remote video screen, with accompanying processing. They can 
communicate with each haptic processor, which take in data from the 
physical parameters of the sense being displayed, e.g. position of the fingers 

20 related to force feedback. Each haptic processor applies the appropriate 
feedback, and communicates its part of the world view to the other 
processors. The communication channels can be fibre optic, infrared, 
wired, or whatever medium is appropriate to the twin demands of bandwidth 
and portability. Adding another sense is as straightforward as adding 

25 another sense module up to the limitations of inter-module bandwidth. 
System autoconfiguration is a straightforward function of the softwarej* 
Controllers for the modules could be general purpose digital signal 
processors for maximum flexibility or specialized processors, e*g. 
application specific integrated circuits, for minimum cost. 
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Vision sense information is provided to the operator through a set of 
goggles containing vision displays . Such a device is the Vision Immersion 
Module manufactured by Kaiser Electro-Optics in Carlsbad', CA 92008. 
Sound may also be provided to the operator through a set of headsets. 
5 , , ' • 

Portable System 

The complete system is worn by the human operator through a 
backpack mechanism which also contains the electronic and power systems. 
The resulting system is a self-supporting exoskeleton; requiring no precise 
10 alignment to conform to the anatomical movement of the arm, and is thus 
comfortable to wear. The device is comfortable to use because it applies 
normal forces to the arm; no twist, no shear to skin or bones* It is also-easy 
to don and doff because entire mechanism is supported through the 
backpack. 

15 The complete system communicates to die world modeler through a 

cable which also provides power to the system. The system x^n alsp be self 
contained by providing power through batteries and communications through 
an RF or IR link. 

The techniques of haptic compression, combined with modern power 

20 electronics and digital signal processing , make it possible to construct a 

totally portable haptic display system. Such a system can be used to explore 
virtual worlds in architectural applications, control remote mobile robots in 
space or nuclear applications, or transverse any number or databases to 
explore them through haptic feedback. 

25 While the invention has been particularly shown and described with 

reference to die preferred embodiments thereof, it will be understood by 
those skilled in the art that various changes in form and detail may be made 
therein without departing from the spirit and scope of the invention. 
What is claimed is: 
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1 . A system for sensing and displaying sensory information to a user 
comprising: 

an world modeler adapted to receive first sensory ifrformation and 
transmit second sensory information, 
5 a communications medium adapted to*carfy the first sensory 

information and the second sensory information, 

a haptic system adapted to sense the first sensory information and 
transmit the first sensory information on the communications medium 4o the 
environment creator, adapted to receive the second sensory information 
10 transmitted on the communications medium by the environment creator and 
further adapted to display the second sensory information, 

whereby the system senses sensory information from the user and 
displays sensory information to the user. 

15 2. the system of claim 1 wherein the world modeler is a computer 

system, 

the computer system adapted to create a mathematical model of an 
environment, 

the computer system further adapted to manipulate the mathematical 
20 model of an environment based upon the received first sensory information, 
the computer system further adapted to create the second sensory 
information based upon the mathematical model of an environment and the 
first sensory information, 

whereby the computer system creates an artificial environment for the 
25 user to interact. 

3 . The system of claim 1 wherein the world modeler is a computer 
system and a remote input-output device, 

the remote input device adapted to sense third sensory information 
30 from the remote input-output device, 
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the computer system adapted to manipulate the third sensory 
information to create the second sensory information, 

the computer system further adapted manipulate the first sensory 
information to create fourth sensory information, 
5 the remote input-output adapted to respond to the fourth sensory 

information, 

whereby the remote input-output device is interactively controlled 
through the computer system by the user. 

10 4. The system of claim 1, 2 or 3 wherein the haptic system 

comprises: 

at least one sensory module adapted to: 
sense direct sensory information, 

process the direct sensory information into the first sensory 
15 information, and 

transmit the first sensory information, 
whereby the sensory module communicates the direct sensory 
information processed by the sensory module to the world modeler. 

20 5. The system of claim 4 wherein the sensory module is further 

adapted to: 

receive the second sensory information, 
process the second sensory information into direct display 
information, and 
25 display the direct display information^ 

whereby the sensory module displays the direct display information 
processed by the sensory module to the user . 



27 



WO 95/20788 



PCT/US9S01227 



Fig. 1 



Environment 
Creator 



Communication* Link 
Sense and Display Information 



12 



37 



Haptie f 
System 



10 



14-^ 



Sense 



Display 



1S 



20 



1/7 



WO 95/20788 



PO7iJS9S/01227 




la/7 



i 



WO 95/20788 PCT/US95/01227 



FIG. 3 



14- 



HAPTIC SYSTEM 



SENSORY 
MODULE 
'A' 




) 



SENSORY 
MODULE 




SENSORY 
MODULE 
*B' 



)/ SENSORY \ 



FIG. 4A 




CONTROLLER 



SENSE UNIT 



COMMUNICATIONS 
UNIT 




FIG. 4B 



FIG. 4C 




CONTROLLER 



SENSE UNIT A 



SENSE UNIT 6 



DISPLAY UNIT 



COMMUNICATIONS 
UNIT 



30C 





CONTROLLER 

DISPLAY UNIT 

COMMUNICATIONS 
UNIT 



30b 




FIG, 5 



REMOTE 
INPUT- 
OUTPUT 



TV 



CONTROL 



28 



SENSE 

V 



ENVIRONMENT 
CREATOR 

^ 



10 



25 




12 



COMMUNICATIONS LINK/ 
COMPRESSED HAPTIC . -} 



14 



52 



INFORMATION 



\ 



HAPTIC 
SYSTEM 



SENSE 



2/7 54-^ 



OISP4JW 




WO 95/20788 



PCT/US95/01227 




SU8STITUTC SHEET (flULE 26) 



PCT/DS9S/DI227 




FIG. 7 



LOCAL PROCESSOR 



BIT3 l/F 



35 



SPIRIT30 
DSP 



IP CARRIER 



AOC/DAC 



DAC 



-83 



ENCODER 



81 



62 



84 



FKJ. 8 
4/7 • 



WO 95/20788 



PCT/US95/01227 



< jl w 
uj O </) 

< 



o ~ 



ui 



S5o< f 

ill J 



2: ^ o 
< o o 
x a. o 



oo 

S3 

o z 



T 



° CD < 

3 12 — 

£C>g 

2oO 
< 



I 

to 



V) 



ui CD — no: 
flc O u. 



(0> 
o = < 

UJ 01 CL 
CD ^ 

o 



< H- 
o 



mo 
o 
I 

o 



o 



< 

Oi 

►—I 

(2h 









t 

o 




.• 

o 









5/7 

SUBSTITUTE SHEET (RULE26) 



WO 95/20788 



PCT/US93/01227 









in 


















(/) 






LJ 




a: 






o 




O 






o 










o 




o 






z 

UJ 














_» 





01 

a 
z 

O 2 

go 
2 o 



a: (/> 
uj o 
o z 

p 

Z LJ 

uj q: 



lo 
IP 



!> 

I to 

Is 

1 ° 
° 

.J" 



< 

52 



< 



gi3 
g 2 



Sow 

l*J o z 

0£ O 

> UJ 
ZQC O 

O ^ 



m 

UJ 

o 
z 



0. 

o 





^ a. uj 

25 ui 
5: m z 



ji 

> . s < > 



mUl€26) 



WO 95/20788 



PCT/US9S/0H27 




FIG. 10 



111 



INTERNATIONAL SEARCH REPORT 



Into jntl Application No 

PCT/US 95/0.1227 



A. CLASSIFICATION OF SUBJECT MATTER 

IPC 6 G06F3/00 



According to International Patent gasification (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 



Minimum docmncntanon tearchcd (classification system followed by ciasnfication symbols) 

IPC 6 G06F 



Documentation searched other than minimum documentation to the extent that such documents arc i ncl u ded in the Odds searched 



Electronic data base consulted during the international search (name of data base and, where practical, search terms used) 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category * Citation of document, with indication, where appropriate, of the relevant passages 



Relevant to claim No. 



WO, A, 92 05519 (ADVANCED ROBOTICS RES) 2 
April 1992 

see page 9, line 14 * page 10, line 32 

SYSTEMS & COMPUTERS IN JAPAN, 

vol. 23, no. 112, 1992 NEW YORK US, 

pages 44-54, XP 000380344 

SATO M. ET AL 'Space Interface Device for 

Artificial Reality - SPIDAR' 

see page 46, left column, line 8 - page 

47, left column, line 8; figure 2 

-/-- 



1-5 



1.4,4,* 



13 



Further diKunrinfr art listed m the contimiauon of box C 



m 



Patent family members are listed in annex. 



* Special categories of cited documents : 

# A* document defining the general state of the art which is not 
considered to be of particular relevance 

"E* earlier ofccsmm out published o^ 
filing date 

*L" rtocumtnt which may throw doubts on priority daim(r) or 
which is cited to establish the pubucation date of another 
citation or other special reason (as specified) 

"O* doetmiettt referring to an oral d isclosu re, use, cifrirrf t io nor 



"T" Isfr puMiAed after the internatianal filma data 

or priority date and not in conflict wim the apriicahotrbut 
cited to understand the principle or theory underlying the 



"P* document published prior to the interiuuioiuU Cling date but 
later than the priority date claimed 



-X* document of particular rdeVance; the daimcd tnvqstfcon 
cannot be considered novel or earom be eonsiclercd hr 
involve an inventive step when the document u tasim alone 

"Y* doctimemofparacuto 

cannot be cu uMde itd to involve an inventive step when me 
document is combined with one or mc« «hex sucn docu- 
ments, such romrnnsnnn being obvious to a person sktOed 
in the art. 

'A' document member of doe same patem tonUy 



Date of the actual completion of the internattonal search 



19 May 1995 



r*f wailing **f tfc^ '»"Tti*1*Ttt l tefaet 

I 9. 05.95 



Name and mailing address of the ISA 

European Patent Office, P.B. SSI 8 Patentlaan 2 
NL-22S0HV Rijswijk 
TcL (+ 31-70) 340-2040, Tx. 31 6S1 coo xd, 
Fax (+ 31-70)340-1016 



Authorized officer 

Bail as > A 



pom PCf /ISA/210 (second sheet) (July IMS) 



page 1 of Z 



INTERNATIONAL SEARCH REPORT 



lam joal Application No 

PCT/US 95/01227 



C(Continuauon) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category " Gtanon of document, with indication, where appropriate, of the relevant passages 



Relevant to claim Now 



IEEE VIRTUAL REALITY ANNUAL SYMPOSIUM , 
September 1993 SEATTLE, WA, 
pages 263-270, XP 000457696 
SHIM0GA K. B. 'A Survey of Perceptual 
Feedback Issues in Dexterous 
Telemanipulation : Part I. Finger Force 
Feedback 1 

see page 264, left column, line 1 - line 
22; figure 1 

IEEE VIRTUAL REALITY ANNUAL SYMPOSIUM, 
September 1993 
pages 271-279, XP 000457697 
SHIM0GA K. B. 'A Survey of Perceptual 
Feedback Issues in Dexterous 
Telemanipulation : Part II. Finger Touch 
Feedback 1 

see the whole document 



1-5 



1-5 



Ftem PCT/ISA/218 (continuation of second ihetl) (July 19M) 



page 2 of Z 



INTERNATIONAL SEARCH REPORT 

[nfoimtaon on patem fiimly members 



Intb aoaal Application No 

PCT/US 95/01227 



Patent document 
cited in starch report 



Publication 
date 



Patent family 
members) 



Publican a 
date 



WO-A-9205519 



02-04-92 



AU-A- 
H3B-A- 



8657391 
2264371 



15-04-92 
25-08-93 



Ford PCT/ISArtlO (paicat Oral* ubu) (July 1992) 



